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ABSTRACT 

In a public school setting administrators are 
frequently under local pressure to make a new project service 
available to all eligible children. However, comparable control 
groups for project evaluation are often absent, and although random 
assignment to treatment groups remains the most systematic 'method of 
providing controls, this is not often possible in the realities of 
operating a big-city school system. Several experimental designs, 
including time-series studies, are suggested as a means of overcoming 
this methodological problem in project evaluation. (DG) 
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One of the essential elements of a good experimental design is random 



assignment of units to treatment groups. Randomization according to Campbell, 
’’renders implausible innumerable rival explanations of observed changes by- 
cutting the lawful relationships which in the natural setting would determine 
which person gets which treatment” (Trismen, 1965). Such a model is represented 
by Design No. 1: 

DESIGN NO. 1: R 0\ x ^2: 

R Oi O2 1 s 



These symbols indicate random treatment assignment of units to two groups, one 
of which receives the treatment. 

In a public school setting, randomization is not that easily attained. 
Administrators responsible for implementing project operations are generally 
under local pressure to make service available to everybody who meets eligibility 
criteria. To convince administrators to do otherwise in the face of such 
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pressure is impossible. But evaluate we must, and some compromise between the 

* 

ideal and the possible has to be made. This presentation attempts to discuss 
what we have been doing because Design No. 1 is not practical in our setting, 
using approaches based on the logic of Campbell and Stanley (1967). 

The simplest standard design is 2: 

DESIGN NO. 2: 0 X X 0 2 

where observations prior to and after application of Treatment X to a- group was 
made. Change scores or post scores of Project children may be compared with 
that of a designated group which may or may not be comparable with the experi- 
mental group. For example, pre-post change of Project children on a Stanford- 
Binet test «may be compared with normal change reflected in the standard error 
of the instrument, which is utilized by test publishers for normative purposes. 
The designated comparison group in such a case is the standardization group. 
This design may enable us to know whether real change had occurred, but is it 
possible to infer that Treatment X was the cause? The Project evaluated may 
be altering schedules or producing changes in participants and staff, which 
could be the true cause of change. The presence of systematic differences 
between the local experimental and the national normative group is one of the 
most plausible rival hypotheses we have to contend with. Some of the tests 
in the market for example, are normed on groups that are not truly representa- 
tive of the population we are comparing it with. 



In che absence of a more appropriate alternative we sometimes utilized ^ 

I 
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Design No, 2 while looking for ways of improving it. Take for example, the 
evaluation of Title I Child Development Project which is referred to as Head 
Start in most places. As with most Title I projects children are enrolled in 
Child Development classes as long as there is available classroom space -- a 
practice which probably prevails in most public school settings. Ideally, random 
treatment assignment could be achieved by denying services to every nth child in 
order to achieve Design No. 1. Such an ideal solution is the practice in most 
university-sponsored preschool programs where research-development and operations 
go hand-in-hand. In a public school setting, where there is a marked break 
between research-development on one side, and project operations on the other, this 
is simply not done. Some compromise had to be reached in order to have a control 

group. We decided to use children on the waiting list to serve as quasi-controls, 

only to discover later that this would not work out. We learned then that chil- 
dren on the waiting list were accepted at any time during the school year whenever 

a vacancy occurred. So we are back to where we were: no controls and a self- 

selected experimental group. 

To resolve the problem of no controls, we planned a two-phase evaluation. 
Phase I utilized Design No. 2, which provided data for required state and federal 
reports. Phase II involved the use of an extended Design No. 2, which could take 
the form of either Design 3 or 4: 

Xi Oj X 2 ^ 0 2 ^ 

°1 x 2 °2 



DESIGN NO. 3: 
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Xj refers to the experiences in Child Development Project. 

X 2 refers to the experiences in different types of Kindergarten programs. 

These designs made possible not only the assessment of preschocV experi- 
ences a year later at entry and at the end of the kindergarten year, but also the 
degree to which such experiences are affected by different kindergarten programs 
such as the expensive and intensive Kindergarten Follow-Through, or state-funded 
Kindergarten Enrichment Project. 

One of the n ew features about Designs 3 and 4 is that it involved the 
use of local control groups, which were missing in Design 2. A more simple illustra 
tion of this model where a local control group is introduced is represented by 
this standard design, Design No. 5: 

DESIGN NO. 5: 0 X X 0 2 

°1 °2 

Instead of a designated standardization group for comparing the local experimental 
group with, now a local control group is introduced, representing an improvement 
over Design 2. One of the obvious advantages of Design No. 5 is the elimination 
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of systematic differences between the treatment and the s t an d ai d 1 z a c 1 on gioups. « 

# # 

t f 

You would notice, however, that the randomization procedure referred to* in 
Design No. 1 i r, missing. 

In working with this design, we have resorted to statistical concrols 
such as the Analysis of Covariance, to equate or adjust for any systematic varia- 
tions between these treatment groups. However, ” adjustment” has limitations in 
terms of imperfect measurement of criterion and covariate, and the infinite number 
of dimensions on which the groups could covary. 

t 

The impracticability of Design No. 1 and the limitations of standard 
Designs 2 and 5', led us to consider the issues of cross-sectional and longitudinal 
studies. V/e decided on time-series studies, extending in time designs presented 



earlier. The essence- 
or subject, before or 

these measurements . 



of time-series designs is periodic measurement on some group 
after the introduction of Treatment or Treatments X into 
1‘his is illustrated by Design No. 6: 



DESIGN No. 6* 



°1 ^ * O 2 O 3 . 0^....0j 

°1 °2 °3 O4 . . . . 0 j 



The 1968 - 1969 evaluation of state-funded Kindergarten Enrichment Project 
represents an application of this model. This Project has identifiable unique 
components designed to provide continuity of enriching educational experiences 



initiated at the Title I Child Development preschool project. This Project 



operated at 31 Title I target schools which has been designated on the basis of 
high percent of families on public assistance, commonly referred to as povert y 

index . 
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Schools with high poverty indices are assigned target status, which 
means eligibility for federal-and state-funded special projects. Poverty indices 
of the Title I target schools ranged from 56^ to 22%, while those of Title I non- 
target schools ranged from 37% to 17%, Inclusion of these schools with lower 
poverty indices among the target schools was made at the expense of other schools 
with higher indices to accommodate racial balance of target status designation. 

The ten experimental schools selected from Title I target schools 
reported poverty indices ranging from 39% to 29%. The selection of these experi- 
mental schools was based primarily on the comparability of their poverty indices 
with those of the control schools selected from the Title I non-target schools. 
Poverty indices of these control schools ranged from 35% to 23%. After designating 
the experimental and control schools, experimental and control children were 

randomly sampled from these schools. 

Thus, designation of these treatment groups was based on an explicit 
criterion of poverty index. The randomization procedure was involved only in 
sampling subjects after treatment groups were selected. Questions of comparability 
of groups, and internal validity of findings could easily be invoked to the extent 
that random assignment of treatment groups was not made. However, the longitudinal 
focus of this evaluation allows us to v/ork around the issue of randomization. 

The study represents a simultaneous evaluation of two projects. Child 
Development and Kindergarten Enrichment, using either Design 7 or 8: 
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0, 



0, 



0, 



OyJ . , . .0. 



DESIGN NO. *8 : 



0- 



°i 



0- 



0, 



X 2 

x 2 



°2 

0 2 



0- 



Oj Ojj • . < • 0 j 

0 3 0^| > » » « 0 j 



0- 



0: 



0/ .... 0; 



0/ . . . . 0. 



Design No. 8 was used because it allowed for the assessment of main effects which 
could be attributed to Kindergarten Enrichment and Child Development experiences, 
and interaction effects of these two factors.* Continuing assessment of these 
children, now in first grade are kept up in terms of yearly administration of 
the same test instruments, follow-up of attendance data, mobility rate or transfers 
to different schools, and records of participation in different projects. 

One of the problems in time ..series studies is the occurrence, of 
."multiple treatment interferences” which appear to be a great source of compound- 
ing and contaminating errors. This problem became very real to us when we 
initiated this year to follow-up the 1965 Child Development Project participants 
on attendance and test- data accumulated over five years from the city-wide test- 
ing program. Older siblings of participants were selected to serve as controls, 
Evaluation Design 9 was used: 



DESIGN NO. 9 : 
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As v/c traced records of the experimental children who tire presently 

t 

at fourth grade bach to kindergarten, v/c noted their exposure to multiple X 
treatments, as a result of increasing use of federal funds in the schools. Thus, 
v/hat had started as an assessment of the sustained value of one project five 
years later, was expanded to include other projects and their obvious interactions. 
Assuming that school records have some degree of reliability which could give us 
the list of X treatments for any given child we anticipate a problem which could 

• affect the internal validity of our re.sults, We are faced with the problem ox ^ 

. . > 

identifying which of the X treatment or treatments did in fact wake the real differ- 
ence in the children’s performance. From the point of external validity, the 4 

' i 

* 

results will be unique to the population, subject of course to replication of 
this kind of assessment. 

Time-series studies are no doubt cumbersome and messy. It might even 
be unmanageable in the absence of a computerized data bank on the pupils. Its 
appropriateness, however, in a school setting where attendance and other school 
records are kept, and annual achievement and aptitude tests constitute such a 
natural part, of the school environment, cannot be denied. 

The extended and modified designs we have presented are part of our 
local efforts to come up with workable evaluation models in a natural sotting 
where ’’control” is an elusive concept. The luxury of providing a ’’control” to 
natural conditions is impossible for being natural and real is antiethical to 
’’control” (Guba, 1969). If evaluation design* are to be workable in our- present 
setting, we need to build upon and modify our present knowledge of the classical 
experimental research paradigm. 
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